Method and system of camera focus for advanced driver assistance system (adas)

ABSTRACT

A camera and production technique for an ADAS. The camera lens and image sensor are positioned along multiple axes using a target located at a first distance from the lens to establish a first relative position between the lens and the image sensor. The first relative position between the lens and the image sensor is modified by a predetermined amount for an object located a second distance from the lens. The second distance is larger than the first distance.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/106,328, filed Jan. 22, 2015, the entire content of which isincorporated herein by reference.

BACKGROUND

Camera-based Advanced Driver Assistance Systems (ADAS) are well known,widely used mass produced machine vision systems useful for a variety ofapplications. Such applications may include, for example, lane departurewarning, lane keeping, vehicle detection, forward collision and adaptivecruise control, and other known driver assistance applications. Withadvances in ADAS applications, greater demands are being placed on themodeling of radial distortion of lenses used in ADAS.

The importance of modeling lens distortion in photometry is described inBrown 1971, in which a distortion model is described (see Duane C.Brown, Close Range Camera Calibration, Photogrametric Engineering 1971).Various known methods exist for determining the radial distortionparameters of a camera after production. For example, Wang et. al. 2009describe an efficient post-production method for calibrating the lensdistortion parameters including the center of radial distortion (seeAiqi Wang, Tainshiang Qiu and Longtan Shao, A Simple Method of RadialDistortion Correction with Center of Distortion Estimation, J. Math.Imaging Vis. 2009). They use images of straight lines, and thus requirea separate calibration step for each camera after production. The methodof Stein (CVPR 1997) can be used to calibrate the lens distortion onlineusing point correspondences, but this method requires non-linearoptimization over four parameters (including K1, K2 and center of radialdistortion) which may lead to stability problems (see Gideon P. Stein,Lens Distortion Calibration Using Point Correspondences, In Proc. CVPR1997).

To some extent nominal lens values may be used for modeling distortion,but this may lead to inaccurate measurements since the center of radialdistortion can vary considerably. One known alternative is to designlenses with minimal distortion so the lenses can be accurately modeledusing the pinhole camera model. However, minimizing distortion in suchdesigns often comes at the expense of the lens MTF, F#, size and lensprice, and may lead to suboptimal performance.

Camera systems having a mechanical or electronic focus may reduce oreliminate the need for accurate focusing during manufacture. However,mechanical focus is not common in ADAS systems, where high reliabilityand long hours of use make mechanical systems impractical. Digital focustechniques, such as those used in some smartphone cameras, typicallyrequire high resolution and small pixel size and often fail to providethe light sensitivity desirable for ADAS. Thus, especially for camerasin ADAS, there is frequently a need for some form of lens focusingduring manufacture.

It is further known that, in most cases, the camera is typically focusedat infinity. However, in practice, it is generally not possible toposition a focus target far enough away to give an ideal focus atinfinity. A typical solution is to use a collimator lens with a targetplaced at a focal length corresponding to the focal length of thecollimator lens. This known method generally requires a very highquality collimator lens to fill the wide FOV. This known method alsogenerally requires precise positioning of both the target and the camerarelative to the collimator. Furthermore, the lens distortion of thecollimator may be compounded with the lens distortion of the cameralens, thus making it difficult to estimate the parameters for the cameralens alone.

SUMMARY OF THE DISCLOSURE

Disclosed herein are methods and systems for camera production in whichthe lens focus procedure can set the location of the center of radialdistortion to a fixed location that may be common to all manufacturedcameras of this type. The technique is well-suited for production ofADAS cameras. In accordance with one embodiment disclosed herein, acamera may be manufactured such that the center of radial distortion isat a set point in the image, and preferably at the center of the image.Once the center of radial distortion is known, the nominal values for K1and K2 may provide good estimates of the true radial distortion. If moreaccuracy is desired, several known methods can also be used (includingthose disclosed in Stein 1997), but with fewer unknown parameters thesolution may tend to be more stable.

In a multi-camera system, if at least one of the cameras has been wellcalibrated for lens distortion in accordance with the present disclosurethen the other lenses may be calibrated online in a straightforward andstable manner.

In many high-end ADAS multi-camera systems there is a main camera withhorizontal field of view (HFOV) between 40 and 70 degrees. This cameracan support many single camera ADAS functions and is often of sharedspecifications with monocular ADAS systems. The other cameras may be ofsignificantly wider or narrower FOV. It is possible to manufacture themain camera with known center of radial distortion and manufacture theother camera or cameras with less strict manufacturing tolerances.

The proposed solution positions a focus and calibration target at afinite distance from the lens, such as 0.5 m. The camera is focused tothat distance by adjusting the position of the lens in five degrees offreedom (three degrees of translation and two degrees of rotation, thelens is rotationally invariant). After optimal focus is achieved, thelens is shifted by a fixed distance to give optimal focus at infinity.Since there is no collimator lens the distortion of the focus at thecalibration target is due to the camera lens only. This process will bedescribed in more detail in the following section.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows one example of the relative positon of a camera lens and animage sensor in one embodiment.

DETAILED DESCRIPTION

In the following description of examples, reference is made to theaccompanying drawings which form a part hereof, and in which it is shownby way of illustration specific examples that can be practiced. It is tobe understood that other examples can be used and structural changes canbe made without departing from the scope of the disclosed examples.Furthermore, while example contexts in which the disclosure can bepracticed are provided, they are not meant to limit the scope of thedisclosure to those contexts.

Camera manufacturing generally involves the production of an assemblythat includes a lens and an image sensor. Production of good qualityimages by the camera typically depends on the lens and the image sensorbeing optimally positioned relative to each other. The term “align” isoften used to describe optimizing the relative position between thecamera lens and the image sensor.

Cameras may be aligned by active control of the relative position of thelens and the image sensor. The alignment process may use motorizedactuators that change the relative position of the lens and the imagesensor, including automatically deciding when a desired relativeposition is achieved.

With reference to the example shown in FIG. 1, the relative movementbetween the lens and the image sensor in this example may be describedin terms of orthogonal axes x, y, z, where:

x represents an axis parallel to the image sensor pixel rows;

y represents an axis parallel to the image sensor pixel columns;

z represents an axis perpendicular to the image sensor plane;

θx represents rotation about the x axis;

θy represents rotation about the y axis;

In one embodiment computer controlled positioning motors may be used toposition the image sensor (and, optionally, a camera housing) relativeto a camera lens that is in a fixed position. In an alternativeembodiment a lens may be moved relative to an image sensor (and,optionally, a camera housing) that is fixed in position. In oneembodiment the focus and calibration target may include a checkerboardpattern having lines that are not straight but that are designed toappear almost straight in the image.

Multiple Axis Alignment to Far Distances without Intermediary Optics

In one example, the camera lens may be initially aligned along multipleaxes using a focus and calibration target positioned at a first,relatively short, distance from the lens (for example, the firstdistance may be on the order of about 0.85 m). The lens orientationalong the θx and θy axes and the distance between the lens and the imagesensor along the z axis may be adjusted to achieve optimal focus overthe entire field of view. The center of radial distortion may bedetected and the lens and image sensor may be aligned so that the centerof radial distortion coincides with a specified point. In a preferredembodiment the specified point coincides with the center of the imagesensor.

Next, in order to change the alignment to one that is optimal forobjects at a second, farther distance, the relative position of thecamera lens and the image sensor along the z axis may be modified by apredetermined amount. In one example, the predetermined amount maydepend on (1) the lens being focused, (2) the near target distance and(3) the desired distance of the optimal object plain. For example, in anembodiment using a Sunny 4067 lens having an Effective Focal Length(EFL) of 5.47 mm, the modification along the z axis is 35 um.

Once the lens and image sensor are aligned the lens may optionally befixed in position relative to the camera holder by any number of knownmethods. For example, the lens may be glued in place into the holder.

In one embodiment the focus and calibration target may preferably coverthe whole image, so that optimal focus can be determined for the wholefield of view. In one embodiment the focus and calibration target may bea checkerboard pattern that has been warped to match the radialdistortion of the lens. In this embodiment the focus and calibrationtarget lines are curved so that they appear almost straight in theimage. This technique may simplify the task of finding a grid pattern inthe image. Furthermore, it may enable more accurate localization of thecorners of the squares in the image, since the corners are almost rightangles. MTF calculating algorithms such as sfrmat2 generally requireslanted lines, so the focus and calibration target in accordance withthis embodiment has a small rotation and the edges are neither exactlyhorizontal nor vertical. Since the checkerboard lines are almoststraight, this angle is almost uniform over the image, improving thereliability and uniformity of sfrmat2.

A fiducial point may be used, for example in the form of a white dot onone of the squares, to help correctly identify points in the image tocorresponding points in the focus of a calibration target. The center ofradial distortion can then be determined from the corresponding targetand image coordinates. The mapping from target plane to image may becharacterized as a homography. Deviations from the homography are due tothe radial distortion, and a closed form solution for the center ofradial distortion is known in the art.

In the above-described embodiment the lens and the image sensor may beinitially aligned using a target positioned at a first, relatively shortdistance from the lens. Then the relative spacing between the lens andthe image sensor may be changed by a predetermined amount. Thistechnique can help to eliminate the need for intermediate optics (suchas a collimator) that would otherwise be positioned between the cameralens and the target in order to image a near target to far distance forallowing camera focus to a far distance. Intermediate optics mayintroduce aberrations into the feedback mechanism and make it difficultto determine the center of radial distortion. Intermediate optics mayalso require high positional accuracy relative to the target, the imagesensor and the lens. The embodiments disclosed herein help to eliminatethe disadvantages associated with the use of intermediary optics.

In the above-described embodiment the lens and image sensor may bealigned so that the center of radial distortion coincides with thecenter of the image sensor. This technique may save the requirement forposition accuracy in x and y of the target relative to the camera beingfocused, because the x and y feedback is directly related to the lens,and not to the image of the target that is used by other x and yfeedback methods. This technique may also allow simple and accurateimage distortion correction without having to calibrate the camera usinga complex target and processes.

In practice it is often useful to have the axis of the camera housingaligned with the image coordinate system. For example, if the housing isa rectangle, if the back of the housing is parallel to the target planeand if the camera is located on a line perpendicular to the target planeand passing through the fiducial point, then the fiducial point shouldappear at the center of the image. However, as described with respect tosome of the embodiments above, the lens may have been mounted in thecamera such that the center of radial distortion coincides with theimage center. If the lens is manufactured to a very high tolerance andthe image sensor is mounted very accurately in the housing then the twocriteria will coincide quite well. However, such manufacturingtolerances may tend to make the units very expensive.

One option would be to choose one or the other criteria. An alternative(and in some cases preferable) option would be to have the camerahousing mounted to the lens such that the center of radial distortion isat the center of the image, then releasing the lens from its holder androtating the assembled unit so that the fiducial point coincides withthe center of the image. The unit may then be glued into an outerhousing that is aligned with the target (i.e. an outer housing having aback that is parallel to the target plane).

The outer housing need not be in the form of a rectangular enclosure. Itmay, for example, include only a back, or it may include only one ormore sides, or it may include a combination of a back and one or moresides. The outer housing need not be rectangular in shape but mayinstead assume any shape that enables the camera to be oriented to thetarget. Optionally, an additional axis of rotation could be added aroundthe z axis and a second fiducial point could be added so that a rollbetween the image and the outer housing could be eliminated.

Although the examples described herein have been described withreference to the accompanying drawings, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosed examples as defined by theappended claims.

1. (canceled)
 2. A camera comprising: a lens with a radial distortion;an image sensor with a plane of sensors; and a housing to maintain aposition between the lens and the image sensor, wherein the position isbased on: an arrangement of the lens and the image sensor based on theradial distortion; an alignment of the lens to the image sensor to focuson a target; and an adjustment of a distance between the lens and theimage sensor, along an axis perpendicular to the plane of the imagesensor, by a predetermined amount following the alignment, theadjustment to focus the lens and the image sensor at infinity.
 3. Thecamera of claim 2, wherein, to make the arrangement of the lens, thelens is moved within at least one of a vertical axis, a verticalrotation about the vertical axis, a horizontal axis, or a horizontalrotation about the horizontal axis, until a center of the radialdistortion coincides with a specified position of the image sensor, bothof the vertical axis and the horizontal axis being parallel to the planeof the image sensor.
 4. The camera of claim 3, wherein the specifiedposition is at the center of the image sensor.
 5. The camera of claim 3,wherein the target includes a feature based on the radial distortion. 6.The camera of claim 5, wherein the feature is a transform of a line isperceived to be the line by the image sensor when subjected to theradial distortion.
 7. The camera of claim 3, wherein the lens isarranged such that the center of the radial distortion coincides withthe center of the target.
 8. The camera of claim 7, wherein the lens isaffixed to the housing, and wherein, to move the lens within at leastone of the vertical axis, the vertical rotation about the vertical axis,the horizontal axis, or the horizontal rotation about the horizontalaxis, until the center of the radial distortion coincides with thespecified position of the image sensor, the lens is: released from thehousing; moved with respect to the housing; and reaffixed to the housingwhen the center of the radial distortion coincides with the specifiedposition of the image sensor.
 9. The camera of claim 7, wherein thetarget includes a fiducial point, and wherein the lens is arranged suchthat the center of the radial distortion coincides with the center ofthe target is based on the fiducial point.
 10. The camera of claim 2,wherein the lens is aligned to the image sensor based on the target bymovement of the lens along the axis perpendicular to the plane of theimage sensor until the entire target is in focus.
 11. The camera ofclaim 2, wherein the predetermined amount is based on an effetive focallength of the lens.
 12. A method comprising: arranging a lens and animage sensor based on a radial distortion of the lens; aligning the lensto the image sensor to focus on a target; and adjusting a distancebetween the lens and the image sensor, along an axis perpendicular to aplane of the image sensor, by a predetermined amount following alignmentto focus the lens and the image sensor at infinity.
 13. The method ofclaim 12, wherein arranging the lens includes moving the lens within atleast one of a vertical axis, a vertical rotation about the verticalaxis, a horizontal axis, or a horizontal rotation about the horizontalaxis, until a center of the radial distortion coincides with a specifiedposition of the image sensor, both of the vertical axis and thehorizontal axis being parallel to the plane of the image sensor.
 14. Themethod of claim 13, wherein the specified position is at the center ofthe image sensor.
 15. The method of claim 13, wherein the targetincludes a feature based on the radial distortion.
 16. The method ofclaim 15, wherein the feature is a transform of a line that is perceivedto be the line by the image sensor when subjected to the radialdistortion.
 17. The method of claim 13, wherein arranging the lensincludes arranging the lens such that the center of the radialdistortion coincides with the center of the target.
 18. The method ofclaim 17, wherein the lens is affixed to a housing that holds the imagessensor, and wherein moving the lens within at least one of the verticalaxis, the vertical rotation about the vertical axis, the horizontalaxis, or the horizontal rotation about the horizontal axis, until thecenter of the radial distortion coincides with the specified position ofthe image sensor includes: releasing the lens from the housing; movingthe lens with respect to the housing; and reaffixing the lens to thehousing when the center of the radial distortion coincides with thespecified position of the image sensor.
 19. The method of claim 17,wherein the target includes a fiducial point, and wherein arranging thelens such that the center of the radial distortion coincides with thecenter of the target is based on the fiducial point.
 20. The method ofclaim 12, wherein aligning the lens to the image sensor based on thetarget includes moving the lens along the axis perpendicular to theplane of the image sensor until the entire target is in focus.
 21. Themethod of claim 12, wherein the predetermined amount is based on aneffective focal length of the lens.
 22. At least one machine readablemedium including instructions that, when executed by a computer, causethe computer to perform operations comprising: arranging a lens and animage sensor based on a radial distortion of the lens; aligning the lensto the image sensor to focus on a target; and adjusting a distancebetween the lens and the image sensor, along an axis perpendicular to aplane of the image sensor, by a predetermined amount following alignmentto focus the lens and the image sensor at infinity.
 23. The at least onemachine readable medium of claim 22, wherein arranging the lens includesmoving the lens within at least one of a vertical axis, a verticalrotation about the vertical axis, a horizontal axis, or a horizontalrotation about the horizontal axis, until a center of the radialdistortion coincides with a specified position of the image sensor, bothof the vertical axis and the horizontal axis being parallel to the planeof the image sensor.
 24. The at least one machine readable medium ofclaim 23, wherein the specified position is at the center of the imagesensor.
 25. The at least one machine readable medium of claim 23,wherein the target includes a feature based on the radial distortion.26. The at least one machine readable medium of claim 25, wherein thefeature is a transform of a line that is perceived to be the line by theimage sensor when subjected to the radial distortion.
 27. The at leastone machine readable medium of claim 23, wherein arranging the lensincludes arranging the lens such that the center of the radialdistortion coincides with the center of the target.
 28. The at least onemachine readable medium of claim 27, wherein the lens is affixed to ahousing that holds the images sensor, and wherein moving the lens withinat least one of the vertical axis, the vertical rotation about thevertical axis, the horizontal axis, or the horizontal rotation about thehorizontal axis, until the center of the radial distortion coincideswith the specified position of the image sensor includes: releasing thelens from the housing; moving the lens with respect to the housing; andreaffixing the lens to the housing when the center of the radialdistortion coincides with the specified position of the image sensor.29. The at least one machine readable medium of claim 27, wherein thetarget includes a fiducial point, and wherein arranging the lens suchthat the center of the radial distortion coincides with the center ofthe target is based on the fiducial point.
 30. The at least one machinereadable medium of claim 22, wherein aligning the lens to the imagesensor based on the target includes moving the lens along the axisperpendicular to the plane of the image sensor until the entire targetis in focus.
 31. The at least one machine readable medium of claim 22,wherein the predetermined amount is based on an effective focal lengthof the lens.